Data Description
The file "concrete.csv" contains important material data used in civil engineering. The concerete compressive strength - 'strength', is the target label to be analysed.
Goal
Using the data available in file concrete.csv, apply feature engineering methods to obtain 85% to 95% accuracy (tolerance limit 95% of the time (confidence level).
Resources Available
The data for this project is available in file https://archive.ics.uci.edu/ml/machine-learning-databases/concrete/compressive/
Attributes details:
Independent variables:
- cement
- slag
- ash
- water
- superplastic
- coarseagg
- fineagg
- age - Concrete hardens with time and strength increases. Usually, concrete is tested after 28 days.
Target:
- strength
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib import pyplot
import seaborn as sns
import scipy.stats as stat
from scipy.stats import zscore
from scipy.stats import randint as sp_randint
import statsmodels.api as sm
from sklearn.utils import resample
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn import metrics
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.preprocessing import PolynomialFeatures
from sklearn import metrics
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.ensemble import (RandomForestRegressor, GradientBoostingRegressor,AdaBoostRegressor,BaggingRegressor)
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import VotingRegressor
from sklearn.svm import SVR
from sklearn.neighbors import KNeighborsRegressor
from sklearn import svm
from time import time
sns.set() # Setting Seaborn default styles
Load Concrete Data Frame
In [2]:
concrete_df = pd.read_csv('concrete.csv')
concrete_df.head(10)
Out[2]:
In [3]:
concrete_df.tail(10)
Out[3]:
In [4]:
df = concrete_df.copy() # Take the backup of original copy and keep working on this copy
Observations:
- There are 8 independent attributes ( cement, slag, ash, water, superplastic, coarseagg, fineagg, age) and one dependent variable (strength).
- All the records are numeric.
1. Exploratory Data Analysis (EDA)
- Find the shape of the data, data type of individual columns
- Check the presence of missing values
- Descriptive stats of numerical columns
- Find the distribution of numerical columns and the asssociated skeweness and presence of outliers
- Distribution of categorical columns
1.1 Univarient Analysis
This section involves finding,
- attribute types
- number of records and number of attributes
- Central tendency of attributes
In [5]:
print ('Information of all columns within the dataframe: ')
print (df.info())
print ('************************************************************************************\n')
print('Shape of the attributes: ')
print (df.shape)
print ('************************************************************************************\n')
print('types of the attributes: ')
print (df.dtypes)
print ('************************************************************************************\n')
df.apply(lambda x: len(x.unique())) # To check whether any dependent attributes are categorical
Out[5]:
In [6]:
# Checking Missing values
temp_df = df[df.isna().any(axis=1)].count()
if temp_df['age'] == 0:
print ('No missing values')
else:
print ('There are missing values')
print (temp_df)
# Verify which of the columns has the -Ve values
for col in df.columns.tolist():
number_of_exp_with_neg = df[df[col] <0].shape[0]
if (df[df[col] <0][col].count() == 0):
print ('No -Ve values in attribute: ' + col)
else:
print ('{0} attribute has {1} -ve values'.format(col, str(number_of_exp_with_neg)))
Observations:
- There are no missing values in any of the columns.
- All attributes are continuous type.
- All attributes are float type, except age (age is of integer type).
- None of the attibute has any negative values.
- There are 1030 records in the dataframe.
- No string present for any of the attribute values.
Checking the Central tendency
In [7]:
df.describe().T
Out[7]:
Observations:
- cement, water, coarseagg, fineagg,strength: There is not much difference in mean and median (50%).
- Attributes: slag and ash, age, there is much difference in mean and 50% values, indicating mean > median, so being a right skewness in data.
- Minimum values of slag, ash and superplastic are '0', hoping these values cannot be accepted in estimating the concrete strength.
- Min value should be less than std, however 'cement' has 'min' and 'std' are almost same. And for 'water, coarseagg, fineagg' std is much lesser than 'min. Hence these attributes needs to be deep-dived.
Let's strong the above observations by using visualization techniques.
Checking Skewness and Outliers
In [8]:
# Function is to find the skewness of Quantitative attributes
def findSkewness(quant_value, col):
if (quant_value.loc[.50] - quant_value.loc[.25]) > (quant_value.loc[.75] - quant_value.loc[.50]):
print('\'' + col + '\' is left-skewed')
elif (quant_value.loc[.50] - quant_value.loc[.25]) < (quant_value.loc[.75] - quant_value.loc[.50]):
print('\'' + col +'\' is right-skewed')
else:
print('\'' + col + '\' Attribute data distribution is symmetric')
# Finding the outliers
# IQR = Q3 - Q1
# Q1 − 1.5 IQR or above Q3 + 1.5 IQR.
def findOutliers(quant_value, col):
Q1 = quant_value[.25]
Q3 = quant_value[.75]
IQR = Q3 - Q1
outlier_left = (Q1 - 1.5 * IQR)
outlier_right = (Q3 + 1.5 * IQR)
number_of_outliers = (concrete_df[(concrete_df[col] <= outlier_left) | (concrete_df[col] >= outlier_right)][col].count())
print("Number of Outliers data in", col, ": " + str(number_of_outliers))
df_quant=df[:].quantile([.25,.50,.75])
cols = df.columns.tolist()
for col in cols:
findSkewness (df_quant[col], col)
print('**********************************************************************')
for col in cols:
findOutliers(df_quant[col], col)
Note:</br> None of the attributes are categorical, hence no need of any transformations are not applicable.
Univariant Visualizations</blue>
Points to note:
- This involves, individual attribute (all attributes are continuous in nature) visual distribution inspection.
- Is there any clusters involves in the distribution
- Visual inspection of outliers and skewness
In [9]:
# Now let's plot all continous variables
def univariant_plot(number_of_attrs):
fig, axs = plt.subplots(number_of_attrs, 3, figsize=(20,35))
# Number of columns from the dataset is 9, computing the dataframe index directly using row and col
for row in range(number_of_attrs):
sns.boxplot(df.iloc[:, row], ax=axs[row, 0])
sns.distplot(df.iloc[:, row], ax=axs[row, 1], axlabel=df.columns[row], kde=False)
sns.distplot(df.iloc[:, row], ax=axs[row, 2], axlabel=df.columns[row])
plt.show()
univariant_plot(9)
Inferences:
- From the above individual continuous attributes distribution, looks like there are multiple gaussians present in some of the attributes.
For example:
- slag, superplasting, courseagg and age has intersecting gaussians. And also there is 37% of data from superplastic, and 46% of data from slag are '0's.
- ash: we can visualize 2 gausians, they both can be at 50 value. And also 55% of the records contain '0' values.
- Strength and cement are slightly normally distributed, however not perfectly.
- outliers present for age, superplastic, slag, lineagg, water, strength
Observations on Outliers:
- Assuming huge number of outliers present for age and superplastic attributes.
- ash has minimal level of outliers, where as slag has very minimum outliers.
Let's expedite % of outliers of individual attributes
Checking % of '0's
In [10]:
# Number of records in 'ash' with '0' values
for col in df.columns.tolist():
num_ash_with_0 = df[df[col] == 0].shape[0]
_percentage_of_0s = num_ash_with_0 / df.shape[0]
print (col + ' has ' + str(_percentage_of_0s) + ' % of 0s')
Final Inferences on univarient analysis:
- cement is almost normal, and with littel right-skewness
- Assuming slag has 2 and rightly skewed.
- ash has two gaussians, looks like independent to each other and rightly skewed.
- water has 3 guassians and slighly left skewed.
- superplastic has 2 gaussians which are intersecting and rightly skewed.
- coarseagg has 3 guassians and almost normal.
- fineagg almost looks normal.
- age has multiple guassians and rightly skewed.
- stregth is almost normal.
Number of records with '0' present for slag(46%), ash(55%), superplastic(37%). Hence these records for these attributes can be missing values.
1.2 Bivarient & Multivarient Analysis
Note: Bivarient plots w.r.to target (strength) attribute
In [11]:
# Scatterplot w.r.to target variable
number_of_attrs = 8
def bivariant_plot():
for row in range(number_of_attrs):
sns.jointplot(df.iloc[:, row], df.iloc[:, 8], kind='hex')
plt.show()
fig, axs = plt.subplots(1, 2, figsize=(10,5))
sns.scatterplot(x=df.columns.get_values()[row], y="strength", data=df, ax=axs[0])
sns.kdeplot(df[df.columns.get_values()[row]], df.strength, cmap="Reds", shade=True, shade_lowest=False, ax=axs[1])
plt.show()
bivariant_plot()
Observations:
- Cement Vs strength: There is very little correlation exists. Most of the values located around cement values 100-400 range (same can be visualized by the kdeplot).
- slag Vs strength, ash Vs strength: There is no correlation between slag/ash and target attribute. Most of the records values are '0's.
- Same with water, superplastic, coarseagg, fineagg and age. And also most of the independent attributes values are point to '0'.
In [12]:
# Pairplot, plotting individual attribute w.r.to target
sns.pairplot(df)
Out[12]:
Observations: Not much observation from the above pairplot - in general
- Frankly no correlation between attributes
- Presence of more than 1 guassians in each attribute distribution and also with outliers.
- Many records of slag, ash superplastic, age attributes values are '0's. Assumption those values might got missed.
Analysis of each attribute:
- ash looks like the data is spread into two clusters. 1 cluster with '0' values. If we consider '0' ash values are missing values, then ash dataset is with only one cluster.
- cement has slight +ve correlation with strength.
- water has very little -ve correlation with coarseagg and fineagg.
- From the graph, age distribution can be categorized into multiple groups and find which age group people are more participated into Concrete distribution
In [13]:
for row in range(8):
sns.jointplot(df.iloc[:, row], df.iloc[:, 8], kind='kde')
plt.show()
In [14]:
# corrlation matrix
cor=df.corr()
cor
Out[14]:
In [15]:
#heatmap
sns.set(font_scale=1.15)
plt.figure(figsize=(15, 10))
sns.heatmap(cor, vmax=.8, linewidths=0.01,
square=True,annot=True,cmap="BuPu",linecolor="blue")
plt.title('Correlation between attributes');
Observations:
- cement: this has negative correlation with slag, ash, and fineagg. And 49.7% positive correlation with target (strength).
- slag: this has negative correlation with ash, coarseagg, fineagg. Little positive correlation (13%) with target (strength).
- ash: negative correlation with water. positive correlation with superplastic. Little negative correlation (10%) with target (strength)
- water: strong negative correlation with superplastic, fineagg. ~27% positive correlation with age. Negative correlation with target (strength).
- superplastic: negative correlation with coarseagg. And positive correlation almst 22% and 36% with fineagg and target (strength)
- coarseagg: very light -ve correlation with fineagg and strength.
- fineagg: very less negative correlation with age and strength.
- age: 33% positive correlation with strength.
As none of the correlation is beyond 50% and between 20-30% with +ve/-ve correlation. we cannot drop any of the attributes at present with this analysis, and proceed with existing dataset.
Observations from all types of analysis:
1. Give data analysis:
- There are 8 independent attributes ( cement, slag, ash, water, superplastic, coarseagg, fineagg, age) and one dependent variable (strength).
- All the records are numeric.
- There are no missing values in any of the columns.
- All attributes are continuous type.
- All attributes are float type, except age (age is of integer type).
- None of the attribute has any negative values.
- There are 1030 records in the dataframe.
- No string present for any of the attribute values.
2. Descriptive statistics analysis:
- cement, water, coarseagg, fineagg, strength: There is not much difference in mean and median (50%).
- Attributes: slag and ash, age, there is much difference in mean and 50% values, indicating mean > median, so being a right skewness in data.
- Minimum values of slag, ash and superplastic are '0', hoping these values cannot be accepted in estimating the concrete strength.
- Min value should be less than std, however 'cement' has 'min' and 'std' are almost same. And for 'water, coarseagg, fineagg' std is much lesser than 'min. Hence these attributes needs to be deep-dived.
3. Univariant visualization:
- Looks like there are multiple gaussians present in some of the attributes.
- For example: § slag, superplasting, courseagg and age has intersecting gaussians. And also there is 37% of data from superplastic, and 46% of data from slag are '0's. § ash: we can visualize 2 gausians, they both can be at 50 value. And also 55% of the records contain '0' values.
- Strength and cement are almost normally distributed, however not a perfect normal distribution.
- outliers present for age, superplastic, slag, lineagg, water, strength
4. Outliers:
- Huge number of outliers present for age and superplastic attributes.
- ash has minimal level of outliers, where as slag has very minimum outliers.
Inferences from Visualizations:
1. Each Attribute distribution:
- cement is almost normal, and with littel right-skewness
- Assuming slag has 2 and rightly skewed.
- ash has 2 gaussians (i.e. 2 clusters), looks like independent to each other and rightly skewed. 1 cluster with '0' values. If we consider '0' ash values are missing values, then ash dataset is with only one cluster.
- water has 3 guassians and slighly left skewed.
- superplastic has 2 gaussians which are intersecting and rightly skewed.
- coarseagg has 3 guassians and almost normal.
- fineagg almost looks normal.
- age has multiple guassians and rightly skewed.
- stregth is almost normal.
- Number of records with '0' present for slag(46%), ash(55%), superplastic(37%). Hence these records for these attributes can be missing values.
2. Correlation:
- cement: this has negative correlation with slag, ash, and fineagg. And 49.7% positive correlation with target (strength). Most of the values located around cement values 100-400 range
- slag: this has negative correlation with ash, coarseagg, fineagg. Little positive correlation (13%) with target (strength).
- ash: negative correlation with water. positive correlation with superplastic. Little negative correlation (10%) with target (strength)
- water: strong negative correlation with superplastic, fineagg. ~27% positive correlation with age. Negative correlation with target (strength).
- superplastic: negative correlation with coarseagg. And positive correlation almst 22% and 36% with fineagg and target (strength)
- coarseagg: very light -ve correlation with fineagg and strength.
- fineagg: very less negative correlation with age and strength.
- age: 33% positive correlation with strength.
Technical Summary:
- Many data has outliers, with right/left skewed and missing values (values which are '0').
- Frankly no strong correlation between attributes
- Clustered data present for all attributes.
- cement, superplastic, age has positive correlation with strength. water has negative correlation with strength. Where as other attributes with very less correlation with target attribute.
- cement is the strong predictor amongst all for strength.
- As all attributes are some or the other way correlated, we cannot drop any of the attributes at present with this analysis, and proceed with existing dataset.
Regression visualization among dependent attributes
In [16]:
# Let us draw lmplot to better analyze relationship among independent variables
# lmplot of
# cement Vs {slag, ash, fineagg}
# slag Vs {ash, coarseagg, fineagg}
# ash Vs {water, superplastic}
# water Vs {superplastic, fineagg}
# superplastic Vs coarseagg
# coarseagg Vs fineagg
# fineagg Vs age
indep_attr_dict = {
'cement': ['slag', 'ash', 'fineagg'],
'slag' : ['ash', 'coarseagg', 'fineagg'],
'ash' : ['water', 'superplastic'],
'water' : ['superplastic', 'fineagg'],
'superplastic' : ['coarseagg'],
'coarseagg' : ['fineagg'],
'fineagg' : ['age']
}
df = concrete_df.copy()
for key in indep_attr_dict:
value = indep_attr_dict[key]
graph_len = len(value)
if (graph_len > 1):
fig, axs = plt.subplots(1, graph_len, figsize=(10,5), sharey=True)
for indx in range(graph_len):
sns.regplot(x=df[key], y=df[value[indx]], ax=axs[indx])
else:
sns.regplot(x=df[key], y=df[value[0]])
plt.show()
Observations:
- Regression lines are horizontal for --> water Vs ash, superplastic Vs water. i.e. whatever value ash changes there is no change in superplastic and whatever water value changes, there is no change in superplastic.
All attributes Outliers boxplot
In [17]:
df.boxplot(figsize=(35,15))
Out[17]:
Observations:
- Outliers present for slag, water, superplastic, fineagg, age and strength
- Huge number of outliers for water and age attributes
In [18]:
# age is integer type, which can be grouped.
tmp_df = concrete_df.copy()
print (tmp_df['age'].value_counts())
print (tmp_df['fineagg'].value_counts())
In [19]:
#Checking for missing values
df.isnull().sum()
Out[19]:
Handling Outliers
In [20]:
#Replacing the outliers by median
filled_outliers_df = concrete_df.copy()
for col in filled_outliers_df.columns[:-1]:
q1 = filled_outliers_df[col].quantile(0.25)
q3 = filled_outliers_df[col].quantile(0.75)
iqr = q3 - q1
low = q1-1.5*iqr
high = q3+1.5*iqr
filled_outliers_df.loc[(filled_outliers_df[col] < low) | (filled_outliers_df[col] > high), col] = filled_outliers_df[col].median()
filled_outliers_df.boxplot(figsize=(35,15))
Out[20]:
Observations:
- Outliers got reduced after filling outlier values with median.
- Most of the outliers were replaced. very less number of ourliers present for fineagg and strength, which can be ignored.
- age still shows good number of upper ourliers, because of below records
age number of records
365 14
270 13
360 6
In [21]:
filled_outliers_df['age'].value_counts()
Out[21]:
In [22]:
sns.pairplot(filled_outliers_df)
Out[22]:
Outliers got reduced when compared from the original RAW data, hence proceeding with this filled data 'filled_outliers_df' dataset
However still clusters did not get reduced. Neither strong correlation among independent attributes, nor between independent attribute Vs target (strength).
3. Feature Engineering, Model Building and Tuning
3.1 Scale the attributes
In [23]:
# Divide the data into X and y datasets (Independent attributes --> X, Dependent attribute --> y)
df = concrete_df.copy()
X = df.drop('strength', axis=1)
y = df[['strength']]
In [24]:
# Scale all columns of concrete_df
X_scaled = preprocessing.scale(X)
X_scaled = pd.DataFrame(X_scaled, columns=X.columns)
y_scaled = preprocessing.scale(y)
y_scaled = pd.DataFrame(y_scaled, columns=y.columns)
print ('X_scaled: ')
print (X_scaled)
print ('y_scaled: ')
print (y_scaled)
In [25]:
# Split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_scaled, test_size=0.30, random_state=1)
3.2 Creation of simple linear model
In [26]:
regression_model = LinearRegression()
regression_model.fit(X_train, y_train)
for idx, col_name in enumerate(X_train.columns):
print("The coefficient for {} is {}".format(col_name, regression_model.coef_[0][idx]))
intercept = regression_model.intercept_[0]
print("The intercept for our model is {}".format(intercept))
y_pred = regression_model.predict(X_test)
print('Performance on training data using Linear Regression:',regression_model.score(X_train, y_train))
print('Performance on testing data using Linear Regression:',regression_model.score(X_test, y_test))
acc = metrics.r2_score(y_test, y_pred)
print('Accuracy: ',acc)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
# Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.DataFrame({'Method':['Linear Regression'], 'accuracy': acc},index={'1'})
results = results[['Method', 'accuracy']]
results
Out[26]:
3.3 Ridge Model
In [27]:
ridge = Ridge(alpha=.3)
ridge.fit(X_train,y_train)
for idx, col_name in enumerate(X_train.columns):
print("Ridge model coefficient for {} is {}".format(col_name, ridge.coef_[0][idx]))
y_pred = ridge.predict(X_test)
print('Performance on training data using Ridge Regression:',ridge.score(X_train, y_train))
print('Performance on testing data using Ridge Regression:',ridge.score(X_test, y_test))
acc = metrics.r2_score(y_test, y_pred)
print('Accuracy: ',acc)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
# Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.concat([results, pd.DataFrame({'Method':['Ridge Regression'], 'accuracy': [acc]},index={'2'})])
results = results[['Method', 'accuracy']]
results
Out[27]:
Observation:
- cement, slag, ash, age attributes has high cofficient value than others. Hence assuming they are the strong predictors for the target (strength).
- In these 4 arrtibues cement is the strongest attribute.
- Assuming water, superplastic, coarseagg, fineagg are less significant attributes.
- There is not much difference in accuracy in Linear and Rigge regression models.
Let's check the same using Lasso, where Lasso dops the least significant attributes.
3.4 Lasso Model
In [28]:
lasso = Lasso(alpha=0.1)
lasso.fit(X_train,y_train)
idx = 0
for col_name in enumerate(X_train.columns):
print("Lasso model coefficient for {} is {}".format(col_name, lasso.coef_[idx]))
idx = idx+1
y_pred = lasso.predict(X_test)
print('Performance on training data using Lasso Regression:',lasso.score(X_train, y_train))
print('Performance on testing data using Lasso Regression:',lasso.score(X_test, y_test))
acc = metrics.r2_score(y_test, y_pred)
print('Accuracy: ',acc)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
y_pred = np.reshape(y_pred, (-1, 1))
# Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.concat([results, pd.DataFrame({'Method':['Lasso Regression'], 'accuracy': [acc]},index={'3'})])
results = results[['Method', 'accuracy']]
results
Out[28]:
Observations:
- Lasso dropped ash, coarseagg and fineagg attributes, which are not same from Ridge regression.
- Lasso accuracy got dropped compared to Ridge and Linear Regressions.
3.5 Generating and checking with Polynomial model
In [29]:
poly = PolynomialFeatures(degree = 2, interaction_only=True)
X_poly = poly.fit_transform(X_scaled)
X_train, X_test, y_train, y_test = train_test_split(X_poly, y, test_size=0.30, random_state=1)
regression_model.fit(X_train, y_train)
print(regression_model.coef_[0])
y_pred = regression_model.predict(X_test)
print('Performance on training data using Linear Regression with Polynomial:',regression_model.score(X_train, y_train))
print('Performance on testing data using Linear Regression with Polynomial:',regression_model.score(X_test, y_test))
acc = metrics.r2_score(y_test, y_pred)
print('Accuracy: ',acc)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
# Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.concat([results, pd.DataFrame({'Method':['Linear Regression with Polynomial'], 'accuracy': [acc]},index={'4'})])
results = results[['Method', 'accuracy']]
results
Out[29]:
Observation:
- There is a lot of improvement in accuracy with Polynomial model.
In [30]:
ridge = Ridge(alpha=.3)
ridge.fit(X_train,y_train)
print ("Ridge model Coefficients:", (ridge.coef_))
y_pred = ridge.predict(X_test)
print('Performance on training data using Ridge Regression with Polynomial:',ridge.score(X_train, y_train))
print('Performance on testing data using Ridge Regression with Polynomial:',ridge.score(X_test, y_test))
acc = metrics.r2_score(y_test, y_pred)
print('Accuracy: ',acc)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
# Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.concat([results, pd.DataFrame({'Method':['Ridge Regression with Polynomial'], 'accuracy': [acc]},index={'5'})])
results = results[['Method', 'accuracy']]
results
Out[30]:
In [31]:
lasso = Lasso(alpha=0.01)
lasso.fit(X_train,y_train)
print ("Lasso model:", (lasso.coef_))
y_pred = lasso.predict(X_test)
print('Performance on training data using Lasso Regression with Polynomial:',lasso.score(X_train, y_train))
print('Performance on testing data using Lasso Regression with Polynomial:',lasso.score(X_test, y_test))
acc = metrics.r2_score(y_test, y_pred)
print('Accuracy: ',acc)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
y_pred = np.reshape(y_pred, (-1, 1))
# Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.concat([results, pd.DataFrame({'Method':['Lasso Regression with Polynomial'], 'accuracy': [acc]},index={'6'})])
results = results[['Method', 'accuracy']]
results
Out[31]:
Note:
- From above table, at present any of 4, 5, 6 models are good in accuracy (models built with Polynomial). Hence let's draw confidence levels of these models.
Note:
Except age, all other attributes units are different. Hence all attributes has to be placed in same units. Hence using Z score for scaling.
3.5 Model building with all attributes
In [32]:
# Scaling the dataset using Z-score to keep all attributes units to same
filled_outliers_df_z = filled_outliers_df.apply(zscore)
filled_outliers_df_z=pd.DataFrame(filled_outliers_df_z,columns=concrete_df.columns)
In [33]:
dtree_model = DecisionTreeRegressor()
def build_dtree_model(df_z, attr_tobe_considered):
# Step: 1 - Separating independent and dependent attributes
if (attr_tobe_considered == 'all'):
X = df_z.iloc[:,0:8]
else:
X = df_z.iloc[:, [0, 1, 2, 3, 7]]
y = filled_outliers_df_z.iloc[:,8]
# Step: 2 - Split X and y into training and test set in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 1)
# Step: 3 - Buidling model
dtree_model.fit(X_train , y_train)
# printing the feature importance
print('Feature importances: \n',pd.DataFrame(dtree_model.feature_importances_,columns=['Imp'],index=X_train.columns))
print ('*****************************************************************')
# Step: 4 - Performance computation
y_pred = dtree_model.predict(X_test)
print(X_test)
print('Performance on training data using DT:',dtree_model.score(X_train, y_train))
print('Performance on testing data using DT:',dtree_model.score(X_test, y_test))
acc_DT = metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_DT)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
# Step: 5 - Regression plot between y_test and y_pred (Test Vs Predicted) values
with sns.axes_style("darkgrid"):
j = sns.jointplot(x=y_test, y=y_pred, kind='reg', color='blue')
j.annotate(stat.pearsonr)
plt.show()
return acc_DT
acc = build_dtree_model(filled_outliers_df_z, 'all')
There is a huge difference in performance on trailing and testing. Which indicates overfitting.
In [34]:
#Store the accuracy results for each model in a dataframe for final comparison
results = pd.concat([results, pd.DataFrame({'Method':['Decision Tree'], 'accuracy': [acc]},index={'7'})])
results = results[['Method', 'accuracy']]
results
Out[34]:
Observations:
- cement, age and water are significant attributes.
- ash, coarseagg, fineagg, superplastic and slag are the less significant variable. These attributes will impact less to the target (strength) column.
- There is a overfitting in the model as the dataset is performing 99% accurately in trainnig data and 84% in testing data (accuracy dropped).
3.2 K fold cross validation
In [35]:
def kfold_cross_validation(method, indx, results, dtree_model):
num_folds = 10
seed = 77
kfold = KFold(n_splits=num_folds, random_state=seed)
results1 = cross_val_score(dtree_model,X, y, cv=kfold)
accuracy=np.mean(abs(results1))
print('Kfold Average accuracy: ',accuracy)
print('Kfold Standard Deviation: ',results1.std())
results = pd.concat([results, pd.DataFrame({'Method':[method], 'accuracy': [accuracy]},index={indx})])
results = results[['Method', 'accuracy']]
return results
results = kfold_cross_validation('Decision Tree K fold','8', results, dtree_model)
In [36]:
results
Out[36]:
3.3 Model building by ignoring least Important attributes
In [37]:
df_z_2 = filled_outliers_df_z.copy()
In [38]:
accuracy = build_dtree_model(df_z_2, 'dropping_attr')
In [39]:
results = pd.concat([results, pd.DataFrame({'Method':'Decision Tree drop least attr', 'accuracy': [accuracy]},index={'9'})])
results = results[['Method', 'accuracy']]
results
Out[39]:
There is no improvement in accuracy from training to testing data. Still overfit model.
3.4 DTree Model pruning
In [40]:
# independent and dependent variables
X = filled_outliers_df_z.iloc[:,0:8]
y = filled_outliers_df_z.iloc[:,8]
# Split X and y into training and test set in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 1)
# Regularizing the Decision tree classifier and fitting the model
reg_dt_model = DecisionTreeRegressor( max_depth = 11,random_state=1,min_samples_leaf=5)
reg_dt_model.fit(X_train, y_train)
Out[40]:
In [41]:
print (pd.DataFrame(reg_dt_model.feature_importances_, columns = ["Imp"], index = X_train.columns))
In [42]:
# View a list of the features and their importance scores
tmp_df = concrete_df.copy()
importances = reg_dt_model.feature_importances_
indices = np.argsort(importances)[::-1][:15]
a = tmp_df.columns[:]
print (a)
features= a.drop('strength',1)
#plot it
plt.figure(1)
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), features[indices])
plt.xlabel('Relative Importance')
Out[42]:
Observations:
- ash, coarseagg and fineagg attributes are less important attributes
In [43]:
y_pred = reg_dt_model.predict(X_test)
print('Performance on training data using DT:',reg_dt_model.score(X_train,y_train))
print('Performance on testing data using DT:',reg_dt_model.score(X_test,y_test))
acc_RDT=metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_RDT)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
In [44]:
results = pd.concat([results, pd.DataFrame({'Method':'Pruned Decision Tree', 'accuracy': [acc_RDT]},index={'10'})])
results = results[['Method', 'accuracy']]
results
Out[44]:
3.5 Pruned Decision Tree K fold
In [45]:
results = kfold_cross_validation('Pruned Decision Tree K fold', '11', results, reg_dt_model)
In [46]:
results
Out[46]:
3.6 Pruned Decision Tree by dropping least important attributes
In [47]:
X = filled_outliers_df_z.iloc[:, [0, 1, 2, 3, 7]]
y = filled_outliers_df_z.iloc[:,8]
# Split X and y into training and test set in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 1)
# Regularizing the Decision tree classifier and fitting the model
reg_dt_model = DecisionTreeRegressor( max_depth = 4,random_state=1,min_samples_leaf=5)
reg_dt_model.fit(X_train, y_train)
y_pred = reg_dt_model.predict(X_test)
print('Performance on training data using DT:',reg_dt_model.score(X_train,y_train))
print('Performance on testing data using DT:',reg_dt_model.score(X_test,y_test))
acc_RDT=metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_RDT)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'Pruned Decision Tree drop least attr', 'accuracy': [acc_RDT]},index={'12'})])
results = results[['Method', 'accuracy']]
In [48]:
results
Out[48]:
- Comparing all models, "Decision Tree with kfold" is giving better results.
- Let us explore the data for hidden clusters using KMeans and PCA component analysis.
K Means Clustering
In [49]:
cluster_range = range( 1, 15 )
cluster_errors = []
df = concrete_df.copy()
for num_clusters in cluster_range:
clusters = KMeans( num_clusters, n_init = 5)
clusters.fit(df)
labels = clusters.labels_
centroids = clusters.cluster_centers_
cluster_errors.append( clusters.inertia_ )
clusters_df = pd.DataFrame( { "num_clusters":cluster_range, "cluster_errors": cluster_errors } )
clusters_df[0:15]
Out[49]:
In [50]:
# Elbow plot
plt.figure(figsize=(12,6))
plt.plot( clusters_df.num_clusters, clusters_df.cluster_errors, marker = "o" )
Out[50]:
Observations:
- From above elbow plot, with our visual analysis there are likely 4 or 5 good clusters. Let's analysze with 4 clusters.
In [51]:
df = filled_outliers_df.copy()
concrete_df_z = df.apply(zscore)
# k=4
cluster = KMeans( n_clusters = 4, random_state = 2354 )
cluster.fit(concrete_df_z)
Out[51]:
In [52]:
# Creating a new column "GROUP" which will hold the cluster id of each record
prediction=cluster.predict(concrete_df_z)
concrete_df_z["GROUP"] = prediction
# Creating a mirror copy for later re-use instead of building repeatedly
concrete_df_z_copy = concrete_df_z.copy(deep = True)
In [53]:
centroids = cluster.cluster_centers_
centroids
Out[53]:
In [54]:
centroid_df = pd.DataFrame(centroids, columns = list(df) )
centroid_df
Out[54]:
In [55]:
# Instead of numerical values of centroids, dawa a box plot of centroids and data
concrete_df_z.boxplot(by = 'GROUP', layout=(3,3), figsize=(15, 10))
Out[55]:
- Here, None of the dimensions are good predictor of target attribute.
- For all the dimensions every cluster have a similar range of values except in one case.
- We can see that the body of the cluster are overlapping.
- So in k means, though, there are clusters in datasets on different dimensions. But we can not see any distinct characteristics of these clusters which tell us to break data into different clusters and build separate models for them.
From the previous analysis, among all attributes 'cement' is good predictor. Hence let's visualize clusters within 'cement' attribute.
In [56]:
var = 'cement'
with sns.axes_style("white"):
plot = sns.lmplot(var,'strength',data=concrete_df_z,hue='GROUP')
plot.set(ylim = (-3,3))
Out[56]:
Observation: Body of 4 clusters are overlapping. Hence cannnot separate the data into multiple clusters using KMeans. Hence proceed with PCA analysis with this 4 group.
In [57]:
# Kmeans clustering is not helping at all. Let's try PRINCIPAL COMPONENT ANALYSIS
# Apply PCA for each group
group = []
group.append(concrete_df_z[concrete_df_z['GROUP']==0])
group.append(concrete_df_z[concrete_df_z['GROUP']==1])
group.append(concrete_df_z[concrete_df_z['GROUP']==2])
group.append(concrete_df_z[concrete_df_z['GROUP']==3])
In [58]:
def pca_analysis(indx):
independent_attr = group[indx].drop(['strength', 'GROUP'] , axis = 1)
dependent_attr = np.array(group[indx].pop('strength'))
# pca = PCA(4)
# group_projected = pca.fit_transform(independent_attr) Reason for avoiding this is the PCA will
# automatically convert data to z scores which is already done. Hence doing the steps of PCA one by one
cov_matrix = np.cov(independent_attr, rowvar=False)
np.linalg.eig(cov_matrix)
eigenvalues, eigenvectors, = np.linalg.eig(cov_matrix)
eig_pairs = [(eigenvalues[index], eigenvectors[:,index]) for index in range(len(eigenvalues))]
# Sort the (eigenvalue, eigenvector) pairs from lowest to highest with respect to eigenvalue
eig_pairs.sort()
eig_pairs.reverse() # reverses the sorted pairs from increasing value of eigenvalue to lowest
# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sort = [eig_pairs[index][0] for index in range(len(eigenvalues))]
eigvectors_sort = [eig_pairs[index][1] for index in range(len(eigenvalues))]
tot = sum(eigenvalues)
var_explained = [(i / tot) for i in sorted(eigenvalues, reverse=True)] # an array of variance explained by each
# eigen vector... there will be 4 entries as there are 4 eigen vectors)
cum_var_exp = np.cumsum(var_explained) # an array of cumulative variance. There will be 8 entries with 8 th entry
# cumulative reaching almost 100%
print ('cum_var_exp: ', cum_var_exp)
plt_title = 'GROUP - ' + str(indx+1)
plt.bar(range(0, 8), var_explained, alpha=0.5, align='center', label='individual explained variance')
plt.step(range(0,8),cum_var_exp, where= 'mid', label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.title(plt_title)
plt.legend(loc = 'best')
plt.show()
for indx in range(4):
pca_analysis(indx)
In [59]:
# Now apply and see whether DT with kfold improves the performance
dtree_model = DecisionTreeRegressor()
results = kfold_cross_validation('Decision Tree K fold with PCA','13', results, dtree_model)
In [60]:
results
Out[60]:
There is no improvement in the accuracy, infact 'DT with PCA' accuracy score is dropped compared to DT with kfold'.
Random Forest Regressor
In [61]:
df = filled_outliers_df.copy()
concrete_df_z = df.apply(zscore)
X = concrete_df_z.iloc[:,0:8]
y = concrete_df_z.iloc[:,8]
# Split X and y into training and test set in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 1)
In [62]:
model=RandomForestRegressor()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Performance on training data using RFR:',model.score(X_train,y_train))
print('Performance on testing data using RFR:',model.score(X_test,y_test))
acc_RFR=metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_RFR)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'Random Forest Regressor', 'accuracy': [acc_RFR]},index={'14'})])
results = results[['Method', 'accuracy']]
In [63]:
results
Out[63]:
Observation:
There is an improvement in accuracy score using Random Forest Regressor algorithm.
In [64]:
results = kfold_cross_validation('Random Forest Regressor with Kfold', '15', results, model)
results
Out[64]:
In [65]:
# Gradiant Boosting Regressor
model=GradientBoostingRegressor()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Performance on training data using GBR:',model.score(X_train,y_train))
print('Performance on testing data using GBR:',model.score(X_test,y_test))
acc_GBR=metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_GBR)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'Gradient Boosting Regressor', 'accuracy': [acc_GBR]},index={'16'})])
results = results[['Method', 'accuracy']]
results
Out[65]:
In [66]:
results = kfold_cross_validation('Gradient Boosting Regressor with kfold', '17', results, model)
results
Out[66]:
Ada Boosting Regressor
In [67]:
model=AdaBoostRegressor()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Performance on training data using ABR:',model.score(X_train,y_train))
print('Performance on testing data using ABR:',model.score(X_test,y_test))
acc_ABR=metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_ABR)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'Ada Boosting Regressor', 'accuracy': [acc_ABR]},index={'18'})])
results = results[['Method', 'accuracy']]
results
Out[67]:
In [68]:
results = kfold_cross_validation('Ada Boosting Regressor with kfold', '19', results, model)
results
Out[68]:
Bagging Regressor
In [69]:
model=BaggingRegressor()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Performance on training data using GBR:',model.score(X_train,y_train))
print('Performance on testing data using GBR:',model.score(X_test,y_test))
acc_BR=metrics.r2_score(y_test, y_pred)
print('Accuracy DT: ',acc_BR)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'Bagging Regressor', 'accuracy': [acc_BR]},index={'20'})])
results = results[['Method', 'accuracy']]
results
Out[69]:
In [70]:
results = kfold_cross_validation('Bagging Regressor with kfold', '21', results, model)
results
Out[70]:
KNN Regressor
In [71]:
error=[]
for i in range(1,30):
knn = KNeighborsRegressor(n_neighbors=i)
knn.fit(X_train,y_train)
pred_i = knn.predict(X_test)
error.append(np.mean(pred_i!=y_test))
plt.figure(figsize=(12,6))
plt.plot(range(1,30),error,color='red', linestyle='dashed',marker='o',markerfacecolor='blue',markersize=10)
plt.title('Error Rate K Value')
plt.xlabel('K Value')
plt.ylabel('Mean error')
Out[71]:
In [72]:
#k=3
model = KNeighborsRegressor(n_neighbors=3)
model.fit(X_train, y_train)
Out[72]:
In [73]:
y_pred = model.predict(X_test)
print('Performance on training data using KNNR:',model.score(X_train,y_train))
print('Performance on testing data using KNNR:',model.score(X_test,y_test))
acc_K=metrics.r2_score(y_test, y_pred)
print('Accuracy KNNR: ',acc_K)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'KNN Regressor', 'accuracy': [acc_K]},index={'22'})])
results = results[['Method', 'accuracy']]
results
Out[73]:
In [74]:
results = kfold_cross_validation('KNN Regressor with kfold', '23', results, model)
results
Out[74]:
In [75]:
model = SVR(kernel='linear')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
# performance on train data
print('Performance on training data using SVR:',model.score(X_train,y_train))
# performance on test data
print('Performance on testing data using SVR:',model.score(X_test,y_test))
#Evaluate the model using accuracy
acc_S=metrics.r2_score(y_test, y_pred)
print('Accuracy SVR: ',acc_S)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'SVR', 'accuracy': [acc_S]},index={'24'})])
results = results[['Method', 'accuracy']]
results
Out[75]:
In [76]:
results = kfold_cross_validation('SVR with kfold', '25', results, model)
results
Out[76]:
Ensemeble with KNN Regressor, Linear Regression and SVM
In [77]:
LR=LinearRegression()
KN=KNeighborsRegressor(n_neighbors=3)
SVM=svm.SVR(kernel='linear')
evc=VotingRegressor(estimators=[('LR',LR),('KN',KN),('SVM',SVM)])
evc.fit(X_train, y_train)
y_pred = evc.predict(X_test)
print('Performance on training data using ensemble:',evc.score(X_train,y_train))
print('Performance on testing data using ensemble:',evc.score(X_test,y_test))
acc_E=metrics.r2_score(y_test, y_pred)
print('Accuracy ensemble: ',acc_E)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
results = pd.concat([results, pd.DataFrame({'Method':'Ensemeble with voting', 'accuracy': [acc_E]},index={'26'})])
results = results[['Method', 'accuracy']]
results
Out[77]:
In [78]:
results = kfold_cross_validation('Ensemeble with voting with kfold', '27', results, evc)
results
Out[78]:
Inferences by looking at accuracy of all models:
- From the above table, Random Forest Regressor (89%), Random Forest Regressor k fold(90%), Gradient Boost Regressor(89%), Gradient Boost Regressor k fold(90%), Bagging Regressor(89%), Bagging Regressor kfold(89%) are giving better results as compared to other models.
- As the dataset have different gaussians, apply Bootstrap sampling technique to resampling large numbers of smaller samples of the same size repeatedly - on RFR (Random Forest Regressor) and GBR(Gradient Boost Regressor) models.
In [79]:
concrete_XY = X.join(y)
Bootstrap Sampling with GBR
In [80]:
values = concrete_XY.values
# Number of bootstrap samples to create
n_iterations = 1000
# size of a bootstrap sample
n_size = int(len(concrete_df_z) * 1)
# run bootstrap
# empty list that will hold the scores for each bootstrap iteration
stats = list()
for i in range(n_iterations):
# prepare train and test sets
train = resample(values, n_samples=n_size) # Sampling with replacement
test = np.array([x for x in values if x.tolist() not in train.tolist()]) # picking rest of the data not considered in sample
# fit model
gbmTree = GradientBoostingRegressor(n_estimators=50)
# fit against independent variables and corresponding target values
gbmTree.fit(train[:,:-1], train[:,-1])
# Take the target column for all rows in test set
y_test = test[:,-1]
# evaluate model
# predict based on independent variables in the test data
score = gbmTree.score(test[:, :-1] , y_test)
predictions = gbmTree.predict(test[:, :-1])
stats.append(score)
# plot scores
def plot_score(stats):
pyplot.hist(stats)
pyplot.show()
# confidence intervals
alpha = 0.95 # for 95% confidence
p = ((1.0-alpha)/2.0) * 100 # tail regions on right and left .25 on each side indicated by P value (border)
lower = max(0.0, np.percentile(stats, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(stats, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))
plot_score(stats)
Bootstrap Sampling with RFR
In [81]:
values = concrete_XY.values
# Number of bootstrap samples to create
n_iterations = 1000
# size of a bootstrap sample
n_size = int(len(concrete_df_z) * 1)
rfTree = RandomForestRegressor(n_estimators=100)
# run bootstrap
# empty list that will hold the scores for each bootstrap iteration
stats = list()
for i in range(n_iterations):
# prepare train and test sets
train = resample(values, n_samples=n_size) # Sampling with replacement
test = np.array([x for x in values if x.tolist() not in train.tolist()]) # picking rest of the data not considered in sample
# fit against independent variables and corresponding target values
rfTree.fit(train[:,:-1], train[:,-1])
# Take the target column for all rows in test set
y_test = test[:,-1]
# evaluate model
# predict based on independent variables in the test data
score = rfTree.score(test[:, :-1] , y_test)
predictions = rfTree.predict(test[:, :-1])
stats.append(score)
plot_score(stats)
The bootstrap random forest regressor model performance is between 84%-90.7% which is better than other classification algorithms.
In [82]:
# Divide the data into Train, validation, test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=1)
Steps involved:¶
- Fit the model with Train dataset.
- Validate the model with valid dataset.
- Hypertuning the model using GridSearchCV and RandomizedSearchCV techniques
- Validate with valid dataset.
- Test with Test dataset.
In [83]:
# Step - 1:
rfTree.fit(X_train, y_train)
Out[83]:
In [84]:
# Step - 2:
y_pred = rfTree.predict(X_val)
print('Performance on training data using Random Forest Regressor:',rfTree.score(X_train,y_train))
print('Performance on validation data using Random Forest Regressor:',rfTree.score(X_val,y_val))
accuracy=metrics.r2_score(y_val, y_pred)
print('Accuracy: ',accuracy)
print('MSE: ',metrics.mean_squared_error(y_val, y_pred))
Step - 3: Hyper parameter tuning
In [85]:
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
print(rfTree.get_params())
In [86]:
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
param_grid = {"max_depth": [3, None],
"max_features": ['auto', 'sqrt'],
"min_samples_split": [2, 5, 10],
"min_samples_leaf": [1, 2, 4],
"bootstrap": [True, False]}
Applying RandomSearchCV
In [87]:
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
# run randomized search
samples = 10 # number of random samples
randomCV = RandomizedSearchCV(rfTree, param_distributions=param_grid, n_iter=samples) #default cv = 3
In [88]:
randomCV.fit(X, y)
print(randomCV.best_params_)
In [89]:
randomCV.cv_results_['mean_test_score']
Out[89]:
Inference: Using Randomized Search CV technique to tune the Randon Forest Regressor, giving performance of 88%.
In [90]:
# Computing the performance of tuned model on train.,valid and test datasets.
print('Performance on training data using RandomizedSearch CV on Randon Forest Regressor:',randomCV.score(X_train,y_train))
print('Performance on validation data RandomizedSearchCV on Random Forest Regressor:',randomCV.score(X_val,y_val))
print('Performance on test data RandomizedSearchCV on Random Forest Regressor:',randomCV.score(X_test,y_test))
y_pred = randomCV.predict(X_test)
accuracy=metrics.r2_score(y_test, y_pred)
print('Accuracy: ',accuracy)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
In [91]:
## Performance of the RandomizedSearchCV applied RFR is showing very good results.
## Lets apply kfold
In [92]:
num_folds = 10
seed = 77
kfold = KFold(n_splits=num_folds, random_state=seed)
results1 = cross_val_score(randomCV,X, y, cv=kfold)
accuracy=np.mean(abs(results1))
print('Kfold Average accuracy: ',accuracy)
print('Kfold Standard Deviation: ',results1.std())
In [93]:
results = pd.concat([results, pd.DataFrame({'Method':'RFR by applying RandomizedSearchCV with kfold', 'accuracy': [accuracy]},index={'28'})])
results = results[['Method', 'accuracy']]
results
Out[93]:
Applying GridSearchCV
In [94]:
# run grid search
gridCV = GridSearchCV(rfTree, param_grid=param_grid)
start = time()
gridCV.fit(X, y)
Out[94]:
In [95]:
gridCV.best_params_
Out[95]:
In [96]:
gridCV.cv_results_['mean_test_score']
Out[96]:
In [97]:
# Computing the performance of tuned model on train.,valid and test datasets.
print('Performance on training data using GridSearch CV on Randon Forest Regressor:',gridCV.score(X_train,y_train))
print('Performance on validation data GridSearchCV on Random Forest Regressor:',gridCV.score(X_val,y_val))
print('Performance on test data GridSearchCV on Random Forest Regressor:',gridCV.score(X_test,y_test))
y_pred = gridCV.predict(X_test)
accuracy=metrics.r2_score(y_test, y_pred)
print('Accuracy: ',accuracy)
print('MSE: ',metrics.mean_squared_error(y_test, y_pred))
In [98]:
## Performance of the GridSearchCV applied RFR is showing as overfit. Performing very good at Testing data, and poor in
## validation and test data
## Lets apply kfold
In [99]:
num_folds = 10
seed = 77
kfold = KFold(n_splits=num_folds, random_state=seed)
results1 = cross_val_score(gridCV,X, y, cv=kfold)
accuracy=np.mean(abs(results1))
print('Kfold Average accuracy: ',accuracy)
print('Kfold Standard Deviation: ',results1.std())
In [100]:
results = pd.concat([results, pd.DataFrame({'Method':'RFR by applying GridSearchCV with kfold', 'accuracy': [accuracy]},index={'28'})])
results = results[['Method', 'accuracy']]
results
Out[100]:
Final Summary :
- Given dataset has lot of clusters with less number of records (1030 in number).
- slag, superplasting, courseagg and age has intersecting gaussians. And also there is 37% of data from superplastic, and 46% of data from slag are '0's.
- Not known on the unit measure of each attribute. Hence applied zscore to bring all attributes into same units.
- Many gausians present for all attributes distributions and also many outliers. To bring down the outliers existence %, replaced outliers with median value.
- Among all model performances, Random Forest regressor is performing 91% with 95% of confidence level.
- After applying RandomizedSearchCV and GridSearchCV hyper parameter tuning techniques, experience the same performance values.
Hence my recommendation is - "Random Forest Regressor" Model suits for predicting the 'strength' attribute using Concrete independent attributes.
In [ ]: